{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DNN Autoencoder example\n",
    "\n",
    "The following tutorial demonstrates how to utilize safekit's multivariate DNN autoencoder to perform anomaly detection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import json\n",
    "import sys\n",
    "\n",
    "from safekit.batch import OnlineBatcher, split_batch\n",
    "from safekit.graph_training_utils import ModelRunner, EarlyStop\n",
    "from safekit.tf_ops import join_multivariate_inputs, dnn, multivariate_loss, eyed_mvn_loss\n",
    "from safekit.util import make_feature_spec, make_loss_spec\n",
    "\n",
    "tf.set_random_seed(408)\n",
    "np.random.seed(408)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_list = [100, 50, 25, 50, 100]\n",
    "lr = 5e-3\n",
    "embed_ratio = 0.75\n",
    "min_embed = 2\n",
    "max_embed = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load the JSON file describing the specifications for the data.\n",
    "\n",
    "This JSON file describes a dictionary specifying the number of features in the input data; the categories corresponding to the features; whether the corresponding category is metadata, input, or output; and the indices which map these categories to specific features. This dictionary can later be used to ease interaction with the data when providing it as input to Tensorflow.\n",
    "\n",
    "`datastart_index` specifies where the event counts begin in a single row of features; this is used by the minibatcher to ensure that it doesn't include metadata in the minibatches it produces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataspecs = json.load(open('../safekit/features/specs/agg/lanl_count_in_count_out_agg.json', 'r'))\n",
    "datastart_index = dataspecs['counts']['index'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the data specifications have been loaded, we instantiate a batcher to divide the data into smaller portions. Since our dataset is rather large, we want to provide it to the model in small batches to avoid filling memory. Adjusting the minibatch size may also improve the model's performance. Here, we'll use a batch size of 256 data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = OnlineBatcher('/home/hutch_research/data/lanl/agg_feats/begin_no_weekends2.txt', 256, skipheader=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To put the data into a form that Tensorflow can efficiently process, we use `join_multivariate_inputs`; this function creates placeholders for the input data and defines operations in the Tensorflow graph that extract learned embeddings of categorical features and concatenate these embeddings together with the continuous features in the input. This defines the input that will be used by the DNN autoencoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_spec = make_feature_spec(dataspecs)\n",
    "x, ph_dict = join_multivariate_inputs(feature_spec, dataspecs, embed_ratio, min_embed, max_embed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we instantiate the graph operations comprising the DNN autoencoder with a single call to the `dnn` function, which will return a Tensorflow variable mapping to the last layer of the DNN. This variable will then be used to define the model's loss. We specify the previously-defined input as the model's input, and provide a list which defines the number of hidden nodes at each hidden layer in the network.\n",
    "\n",
    "In addition to model depth and width, this function has additional hyperparameters that can be tuned, including activation function, weight initialization range, and dropout factor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "h = dnn(x, layer_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To determine how to compute losses, we use `make_loss_spec` to generate a specification dictionary mapping loss functions to features in the input. Then, `multivariate_loss` interprets this specification to define operations to compute feature-wise losses according to the data specifications—since the inputs can be a mixture of continuous and categorical features, their losses need to be defined accordingly. This function supports the use of three different loss functions—`eyed_mvn_loss`, `diag_mvn_loss`, and `full_mvn_loss`. Here we use `eyed_mvn_loss` to compute the squared error for predictions. The latter two use either diagonal or full covariance matrices to compute the Mahalonobis distance true values and predictions.\n",
    "\n",
    "Once we define the graph operations that compute the squared error loss, we sum the losses over all features and average these losses over all data points in the minibatch. This is the scalar loss we will attempt to minimize using gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_spec = make_loss_spec(dataspecs, eyed_mvn_loss)\n",
    "loss_matrix = multivariate_loss(h, loss_spec, ph_dict)\n",
    "\n",
    "loss_vector = tf.reduce_sum(loss_matrix, reduction_indices=1)\n",
    "loss = tf.reduce_mean(loss_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To map losses back to our input features easily, we'll next define a function that we can call during the training loop that will write metadata and losses for each data point in the current minibatch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_results(data_dict, feat_loss, outfile):\n",
    "    for d, u, t, l in zip(data_dict['time'].flatten().tolist(),\n",
    "                          data_dict['user'].tolist(),\n",
    "                          data_dict['redteam'].flatten().tolist(),\n",
    "                          feat_loss.flatten().tolist()):\n",
    "        outfile.write('%s %s %s %s\\n' % (int(d), u, int(t), l))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we instantiate a `ModelRunner` object, which provides a simple interface for interacting with the Tensorflow session. Instantiating this object will define the optimizer Tensorflow will use for gradient descent and initialize all of the variables in the Tensorflow graph. We can then use the `train_step` method on this object to perform an optimization step or the `eval` method to retrieve the values of arbitrary tensors in the graph.\n",
    "\n",
    "The `loss_feats` variable specifies the names of the features over which we are computing losses. In this case, it is a single category—the counts of the categorical features—though in instances with a mixture of categorical and continuous features, more than one category could be represented here.\n",
    "\n",
    "In order to record the losses for all of the features, we define a list `eval_tensors` that contains tensors whose values we want to retrieve during training. We'll provide this list to the `ModelRunner`'s `eval` method during the training loop to compute these tensors, then record their values with the `write_results` function defined previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# other args incl. learning rate, optimizer, decay rate...\n",
    "model = ModelRunner(loss, ph_dict, learnrate=lr, opt='adam')\n",
    "\n",
    "loss_feats = [triple[0] for triple in loss_spec]\n",
    "\n",
    "# list of tensors we want to retrieve at each training step; can also add loss_matrix to this\n",
    "eval_tensors = [loss, loss_vector]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we begin training our model. We start by defining a stopping criteria for training using the `EarlyStop` object; if our model's performance doesn't improve after 20 training steps, the `check_error` function we instantiate will return `False`, and training will be discontinued.\n",
    "\n",
    "Inside the training loop, `split_batch` is first used to construct a dictionary for Tensorflow that maps features to the placeholders used to feed data into the computational graph during training. Since our targets are defined separately from the inputs provided to our batcher, we add these to the dictionary.\n",
    "\n",
    "We retrieve the losses for the current batch, then perform a training step to perform gradient descent over a single batch of inputs. This process repeats until either the batcher has reached the end of the input file, the stopping criteria has been met, or the model's error has diverged to infinity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "index: 256, loss: 7235519488.0000\n",
      "index: 512, loss: 27242684.0000\n",
      "index: 768, loss: 7783619.0000\n",
      "index: 1024, loss: 10031691.0000\n",
      "index: 1280, loss: 5930339.5000\n",
      "index: 1536, loss: 3970240.2500\n",
      "index: 1792, loss: 5967686.0000\n",
      "index: 2048, loss: 4960282.0000\n",
      "index: 2304, loss: 4517553.5000\n",
      "index: 2560, loss: 7036947.0000\n",
      "index: 2816, loss: 2647112.0000\n",
      "index: 3072, loss: 10882352.0000\n",
      "index: 3328, loss: 2459881.0000\n",
      "index: 3584, loss: 1709068.6250\n",
      "index: 3840, loss: 5032989.0000\n",
      "index: 4096, loss: 1138042.7500\n",
      "index: 4352, loss: 1072580.0000\n",
      "index: 4608, loss: 1090286.5000\n",
      "index: 4864, loss: 1288823.0000\n",
      "index: 5120, loss: 977030.6250\n",
      "index: 5376, loss: 1519262.2500\n",
      "index: 5632, loss: 19444328.0000\n",
      "index: 5888, loss: 730013.1875\n",
      "index: 6144, loss: 1931345.5000\n",
      "index: 6400, loss: 616478.6250\n",
      "index: 6656, loss: 537939.9375\n",
      "index: 6912, loss: 446257.3125\n",
      "index: 7168, loss: 458408.2500\n",
      "index: 7424, loss: 228352.2344\n",
      "index: 7680, loss: 231614.3125\n",
      "index: 7936, loss: 1510352128.0000\n",
      "index: 8192, loss: 279777760.0000\n",
      "index: 8448, loss: 12488026.0000\n",
      "index: 8704, loss: 4128352.2500\n",
      "index: 8960, loss: 3906902.2500\n",
      "index: 9216, loss: 6845220.0000\n",
      "index: 9472, loss: 8637100.0000\n",
      "index: 9728, loss: 3381837.5000\n",
      "index: 9984, loss: 4977448.5000\n",
      "index: 10240, loss: 8576134.0000\n",
      "index: 10496, loss: 5675652.0000\n",
      "index: 10752, loss: 5865278.0000\n",
      "index: 11008, loss: 12642810.0000\n",
      "index: 11264, loss: 4010467.0000\n",
      "index: 11520, loss: 10805947.0000\n",
      "index: 11776, loss: 3582368.5000\n",
      "index: 12032, loss: 2689452.2500\n",
      "index: 12288, loss: 2520262.7500\n",
      "index: 12544, loss: 3555353.0000\n",
      "index: 12800, loss: 3503253.7500\n",
      "index: 13056, loss: 3725689.5000\n",
      "index: 13312, loss: 35779532.0000\n",
      "index: 13568, loss: 8105094.0000\n",
      "index: 13824, loss: 1726894.7500\n",
      "index: 14080, loss: 1413744.5000\n",
      "index: 14336, loss: 1358785.2500\n",
      "index: 14592, loss: 1094467.2500\n",
      "index: 14848, loss: 978339.6875\n",
      "index: 15104, loss: 426696.0625\n",
      "index: 15360, loss: 775939.7500\n",
      "index: 15616, loss: 1391880.6250\n",
      "index: 15872, loss: 7020786.5000\n",
      "index: 16128, loss: 661321216.0000\n",
      "index: 16384, loss: 298806016.0000\n",
      "index: 16640, loss: 32990268.0000\n",
      "index: 16896, loss: 3395563.5000\n",
      "index: 17152, loss: 4040128.5000\n",
      "index: 17408, loss: 4486418.0000\n",
      "index: 17664, loss: 4824609.0000\n",
      "index: 17920, loss: 1824382.8750\n",
      "index: 18176, loss: 4657534.5000\n",
      "index: 18432, loss: 5810882.5000\n",
      "index: 18688, loss: 4730784.5000\n",
      "index: 18944, loss: 8039659.5000\n",
      "index: 19200, loss: 2968030.7500\n",
      "index: 19456, loss: 10871150.0000\n",
      "index: 19712, loss: 3624765.7500\n",
      "index: 19968, loss: 5046599.0000\n",
      "index: 20224, loss: 5458169.0000\n",
      "index: 20480, loss: 4934442.0000\n",
      "index: 20736, loss: 5491905.0000\n",
      "index: 20992, loss: 37256656.0000\n",
      "index: 21248, loss: 34254128.0000\n",
      "index: 21504, loss: 1934272.2500\n",
      "index: 21760, loss: 3556305.2500\n",
      "index: 22016, loss: 1468579.6250\n",
      "index: 22272, loss: 1860884.8750\n",
      "index: 22528, loss: 1018711.5625\n",
      "index: 22784, loss: 4953324.0000\n",
      "index: 23040, loss: 568761.0625\n",
      "index: 23296, loss: 435324.4688\n",
      "index: 23552, loss: 1894546176.0000\n",
      "index: 23808, loss: 23740030.0000\n",
      "index: 24064, loss: 4064124.7500\n",
      "index: 24320, loss: 9537622.0000\n",
      "index: 24576, loss: 3094605.5000\n",
      "index: 24832, loss: 5333090.0000\n",
      "index: 25088, loss: 3305423.0000\n",
      "index: 25344, loss: 5604303.0000\n",
      "index: 25600, loss: 9629730.0000\n",
      "index: 25856, loss: 7511816.5000\n",
      "index: 26112, loss: 8963932.0000\n",
      "index: 26368, loss: 7830681.5000\n",
      "index: 26624, loss: 5948107.0000\n",
      "index: 26880, loss: 9500801.0000\n",
      "index: 27136, loss: 4449654.0000\n",
      "index: 27392, loss: 2797184.2500\n",
      "index: 27648, loss: 5868725.0000\n",
      "index: 27904, loss: 5605241.0000\n",
      "index: 28160, loss: 2291230.0000\n",
      "index: 28416, loss: 11034073.0000\n",
      "index: 28672, loss: 66982884.0000\n",
      "index: 28928, loss: 17621452.0000\n",
      "index: 29184, loss: 1405732.8750\n",
      "index: 29440, loss: 2729392.0000\n",
      "index: 29696, loss: 3585086.0000\n",
      "index: 29952, loss: 3314784.2500\n",
      "index: 30208, loss: 941621.9375\n",
      "index: 30464, loss: 2095925.7500\n",
      "index: 30720, loss: 4834960.0000\n",
      "index: 30976, loss: 1533604.0000\n",
      "index: 31232, loss: 955039.1875\n",
      "index: 31488, loss: 1904799.5000\n",
      "index: 31744, loss: 6633335.0000\n",
      "index: 32000, loss: 2024992.2500\n",
      "index: 32256, loss: 1311274.1250\n",
      "index: 32512, loss: 1462599168.0000\n",
      "index: 32768, loss: 302326688.0000\n",
      "index: 33024, loss: 12037528.0000\n",
      "index: 33280, loss: 4540839.5000\n",
      "index: 33536, loss: 3757124.5000\n",
      "index: 33792, loss: 5941743.0000\n",
      "index: 34048, loss: 8267137.0000\n",
      "index: 34304, loss: 3934904.5000\n",
      "index: 34560, loss: 13889314.0000\n",
      "index: 34816, loss: 17438712.0000\n",
      "index: 35072, loss: 16146985.0000\n",
      "index: 35328, loss: 11497622.0000\n",
      "index: 35584, loss: 6303002.0000\n",
      "index: 35840, loss: 6496538.0000\n",
      "index: 36096, loss: 13664460.0000\n",
      "index: 36352, loss: 4814752.5000\n",
      "index: 36608, loss: 4282674.0000\n",
      "index: 36864, loss: 4451230.5000\n",
      "index: 37120, loss: 6432910.5000\n",
      "index: 37376, loss: 3171695.2500\n",
      "index: 37632, loss: 170119104.0000\n",
      "index: 37888, loss: 3440648.7500\n",
      "index: 38144, loss: 7070665.5000\n",
      "index: 38400, loss: 3944027.7500\n",
      "index: 38656, loss: 4049758.5000\n",
      "index: 38912, loss: 2555666.2500\n",
      "index: 39168, loss: 5465524.5000\n",
      "index: 39424, loss: 1670247.5000\n",
      "index: 39680, loss: 2735198.2500\n",
      "index: 39936, loss: 3765357.0000\n",
      "index: 40192, loss: 2481115.5000\n",
      "index: 40448, loss: 1090574.2500\n",
      "index: 40704, loss: 3105685.0000\n",
      "index: 40960, loss: 5034758.5000\n",
      "index: 41216, loss: 3520662.5000\n",
      "index: 41472, loss: 2186295.2500\n",
      "index: 41728, loss: 2995131.2500\n",
      "index: 41984, loss: 1101201152.0000\n",
      "index: 42240, loss: 15912877.0000\n",
      "index: 42496, loss: 6625280.0000\n",
      "index: 42752, loss: 9893288.0000\n",
      "index: 43008, loss: 12584206.0000\n",
      "index: 43264, loss: 16815916.0000\n",
      "index: 43520, loss: 6501371.5000\n",
      "index: 43776, loss: 10813494.0000\n",
      "index: 44032, loss: 18686822.0000\n",
      "index: 44288, loss: 6827920.5000\n",
      "index: 44544, loss: 5983177.0000\n",
      "index: 44800, loss: 6265879.0000\n",
      "index: 45056, loss: 2110364.0000\n",
      "index: 45312, loss: 4813708.0000\n",
      "index: 45568, loss: 2912116.5000\n",
      "index: 45824, loss: 2115858.5000\n",
      "index: 46080, loss: 3860207.7500\n",
      "index: 46336, loss: 4836732.0000\n",
      "index: 46592, loss: 2442341.5000\n",
      "index: 46848, loss: 7745298.5000\n",
      "index: 47104, loss: 80462416.0000\n",
      "index: 47360, loss: 3031795.5000\n",
      "index: 47616, loss: 1430720.2500\n",
      "index: 47872, loss: 1132247.7500\n",
      "index: 48128, loss: 1421044.0000\n",
      "index: 48384, loss: 2194797.5000\n",
      "index: 48640, loss: 901296.8750\n",
      "index: 48896, loss: 2965310.0000\n",
      "index: 49152, loss: 2806859.2500\n",
      "index: 49408, loss: 3387601.5000\n",
      "index: 49664, loss: 901062.6250\n",
      "index: 49920, loss: 1365812.5000\n",
      "index: 50176, loss: 2024214.5000\n",
      "index: 50432, loss: 1279497.7500\n",
      "index: 50688, loss: 1568868.0000\n",
      "index: 50944, loss: 471294.0000\n",
      "index: 51200, loss: 769448064.0000\n",
      "index: 51456, loss: 7596328.0000\n",
      "index: 51712, loss: 12047672.0000\n",
      "index: 51968, loss: 9878874.0000\n",
      "index: 52224, loss: 11995796.0000\n",
      "index: 52480, loss: 21670112.0000\n",
      "index: 52736, loss: 11069196.0000\n",
      "index: 52992, loss: 43333760.0000\n",
      "index: 53248, loss: 37033156.0000\n",
      "index: 53504, loss: 20266298.0000\n",
      "index: 53760, loss: 2711264.0000\n",
      "index: 54016, loss: 5024154.0000\n",
      "index: 54272, loss: 1173702.0000\n",
      "index: 54528, loss: 2189399.0000\n",
      "index: 54784, loss: 5305599.0000\n",
      "index: 55040, loss: 3482091.5000\n",
      "index: 55296, loss: 4167097.7500\n",
      "index: 55552, loss: 8254553.0000\n",
      "index: 55808, loss: 8067438.0000\n",
      "index: 56064, loss: 6240556.0000\n",
      "index: 56320, loss: 184336784.0000\n",
      "index: 56576, loss: 3644127.5000\n",
      "index: 56832, loss: 3124641.5000\n",
      "index: 57088, loss: 1590179.2500\n",
      "index: 57344, loss: 3443346.5000\n",
      "index: 57600, loss: 1817575.2500\n",
      "index: 57856, loss: 3496151.0000\n",
      "index: 58112, loss: 4601976.0000\n",
      "index: 58368, loss: 1510634.5000\n",
      "index: 58624, loss: 3688040.0000\n",
      "index: 58880, loss: 1439799.8750\n",
      "index: 59136, loss: 1741259.5000\n",
      "index: 59392, loss: 1792681.2500\n",
      "index: 59648, loss: 8401995.0000\n",
      "index: 59904, loss: 1799612.2500\n",
      "index: 60160, loss: 1708274.1250\n",
      "index: 60416, loss: 1720829.7500\n",
      "index: 60672, loss: 691035776.0000\n",
      "index: 60928, loss: 10587527.0000\n",
      "index: 61184, loss: 5946347.5000\n",
      "index: 61440, loss: 5983269.0000\n",
      "index: 61696, loss: 11483576.0000\n",
      "index: 61952, loss: 9780460.0000\n",
      "index: 62208, loss: 60102844.0000\n",
      "index: 62464, loss: 26942460.0000\n",
      "index: 62720, loss: 11720822.0000\n",
      "index: 62976, loss: 13462684.0000\n",
      "index: 63232, loss: 2117935.5000\n",
      "index: 63488, loss: 5332001.0000\n",
      "index: 63744, loss: 12916678.0000\n",
      "index: 64000, loss: 11358124.0000\n",
      "index: 64256, loss: 8990565.0000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "index: 64512, loss: 12476050.0000\n",
      "index: 64768, loss: 20791440.0000\n",
      "index: 65024, loss: 31349966.0000\n",
      "index: 65280, loss: 10900652.0000\n",
      "index: 65536, loss: 3343473.0000\n",
      "index: 65792, loss: 9055377.0000\n",
      "index: 66048, loss: 7649208.5000\n",
      "index: 66304, loss: 1584254.1250\n",
      "index: 66560, loss: 2929770.2500\n",
      "index: 66816, loss: 8568593.0000\n",
      "index: 67072, loss: 3307513.2500\n",
      "index: 67328, loss: 2161881.2500\n",
      "index: 67584, loss: 5490057.0000\n",
      "index: 67840, loss: 2327615.0000\n",
      "index: 68096, loss: 1260092.2500\n",
      "index: 68352, loss: 723937024.0000\n",
      "index: 68608, loss: 645145856.0000\n",
      "index: 68864, loss: 13266762.0000\n",
      "index: 69120, loss: 11260218.0000\n",
      "index: 69376, loss: 7383053.5000\n",
      "index: 69632, loss: 9246822.0000\n",
      "index: 69888, loss: 16765551.0000\n",
      "index: 70144, loss: 7445397.0000\n",
      "index: 70400, loss: 17689336.0000\n",
      "index: 70656, loss: 11074226.0000\n",
      "index: 70912, loss: 9646866.0000\n",
      "index: 71168, loss: 4960465.5000\n",
      "index: 71424, loss: 6416443.0000\n",
      "index: 71680, loss: 7880465.5000\n",
      "index: 71936, loss: 9655702.0000\n",
      "index: 72192, loss: 8690001.0000\n",
      "index: 72448, loss: 7868537.5000\n",
      "index: 72704, loss: 5155475.0000\n",
      "index: 72960, loss: 10597140.0000\n",
      "index: 73216, loss: 6608046.0000\n",
      "index: 73472, loss: 8992840.0000\n",
      "index: 73728, loss: 123922136.0000\n",
      "index: 73984, loss: 4897044.0000\n",
      "index: 74240, loss: 4111957.0000\n",
      "index: 74496, loss: 5905786.5000\n",
      "index: 74752, loss: 7360596.0000\n",
      "index: 75008, loss: 18779430.0000\n",
      "index: 75264, loss: 3139640.2500\n",
      "index: 75520, loss: 6584802.0000\n",
      "index: 75776, loss: 4518617.0000\n",
      "index: 76032, loss: 6769235.0000\n",
      "index: 76288, loss: 1708819.7500\n",
      "index: 76544, loss: 3526709.5000\n",
      "index: 76800, loss: 2716873.0000\n",
      "index: 77056, loss: 2425363.2500\n",
      "index: 77312, loss: 6317699.0000\n",
      "index: 77568, loss: 1261412352.0000\n",
      "index: 77824, loss: 172595392.0000\n",
      "index: 78080, loss: 19653682.0000\n",
      "index: 78336, loss: 11162692.0000\n",
      "index: 78592, loss: 18121476.0000\n",
      "index: 78848, loss: 16860432.0000\n",
      "index: 79104, loss: 30205490.0000\n",
      "index: 79360, loss: 7683413.5000\n",
      "index: 79616, loss: 32481542.0000\n",
      "index: 79872, loss: 22591940.0000\n",
      "index: 80128, loss: 15033772.0000\n",
      "index: 80384, loss: 10710663.0000\n",
      "index: 80640, loss: 11355656.0000\n",
      "index: 80896, loss: 9643506.0000\n",
      "index: 81152, loss: 17102620.0000\n",
      "index: 81408, loss: 26694576.0000\n",
      "index: 81664, loss: 16017546.0000\n",
      "index: 81920, loss: 8637132.0000\n",
      "index: 82176, loss: 15429154.0000\n",
      "index: 82432, loss: 8437906.0000\n",
      "index: 82688, loss: 20203016.0000\n",
      "index: 82944, loss: 45479788.0000\n",
      "index: 83200, loss: 7226776.5000\n",
      "index: 83456, loss: 3788436.2500\n",
      "index: 83712, loss: 5453200.0000\n",
      "index: 83968, loss: 4985602.0000\n",
      "index: 84224, loss: 2227507.0000\n",
      "index: 84480, loss: 3935335.5000\n",
      "index: 84736, loss: 3439694.5000\n",
      "index: 84992, loss: 6358977.0000\n",
      "index: 85248, loss: 4024469.5000\n",
      "index: 85504, loss: 2358392.0000\n",
      "index: 85760, loss: 4770649.0000\n",
      "index: 86016, loss: 3631803.7500\n",
      "index: 86272, loss: 4160224.0000\n",
      "index: 86528, loss: 6335092.0000\n",
      "index: 86784, loss: 1363541376.0000\n",
      "index: 87040, loss: 353467232.0000\n",
      "index: 87296, loss: 17394462.0000\n",
      "index: 87552, loss: 8942629.0000\n",
      "index: 87808, loss: 6908833.5000\n",
      "index: 88064, loss: 8808584.0000\n",
      "index: 88320, loss: 13677111.0000\n",
      "index: 88576, loss: 7346116.0000\n",
      "index: 88832, loss: 8568907.0000\n",
      "index: 89088, loss: 5565395.0000\n",
      "index: 89344, loss: 4536313.5000\n",
      "index: 89600, loss: 5638743.0000\n",
      "index: 89856, loss: 4827321.5000\n",
      "index: 90112, loss: 2854233.0000\n",
      "index: 90368, loss: 6252753.5000\n",
      "index: 90624, loss: 6016711.5000\n",
      "index: 90880, loss: 6502366.0000\n",
      "index: 91136, loss: 3520757.5000\n",
      "index: 91392, loss: 3744808.7500\n",
      "index: 91648, loss: 3191787.0000\n",
      "index: 91904, loss: 5250876.5000\n",
      "index: 92160, loss: 14451422.0000\n",
      "index: 92416, loss: 6136116.0000\n",
      "index: 92672, loss: 2672324.0000\n",
      "index: 92928, loss: 2535365.0000\n",
      "index: 93184, loss: 1695857.2500\n",
      "index: 93440, loss: 15368484.0000\n",
      "index: 93696, loss: 8034078.0000\n",
      "index: 93952, loss: 5882827.0000\n",
      "index: 94208, loss: 8781952.0000\n",
      "index: 94464, loss: 3150806.0000\n",
      "index: 94720, loss: 2165770.5000\n",
      "index: 94976, loss: 3815217.0000\n",
      "index: 95232, loss: 19535078.0000\n",
      "index: 95488, loss: 7436737.5000\n",
      "index: 95744, loss: 35513828.0000\n",
      "index: 96000, loss: 2791510.5000\n",
      "index: 96256, loss: 3135267840.0000\n",
      "index: 96512, loss: 12178174.0000\n",
      "index: 96768, loss: 6304775.0000\n",
      "index: 97024, loss: 3943344.7500\n",
      "index: 97280, loss: 6151205.5000\n",
      "index: 97536, loss: 18351722.0000\n",
      "index: 97792, loss: 2735817.5000\n",
      "index: 98048, loss: 3151352.7500\n",
      "index: 98304, loss: 6682843.0000\n",
      "index: 98560, loss: 3772354.0000\n",
      "index: 98816, loss: 4601547.0000\n",
      "index: 99072, loss: 3350843.2500\n",
      "index: 99328, loss: 3262855.5000\n",
      "index: 99584, loss: 5236224.5000\n",
      "index: 99840, loss: 2752206.0000\n",
      "index: 100096, loss: 4457377.5000\n",
      "index: 100352, loss: 1844119.6250\n",
      "index: 100608, loss: 2157898.5000\n",
      "index: 100864, loss: 4001614.5000\n",
      "index: 101120, loss: 10734417.0000\n",
      "index: 101376, loss: 2167117.0000\n",
      "index: 101632, loss: 2637240.0000\n",
      "index: 101888, loss: 8111063.0000\n",
      "index: 102144, loss: 1167697.0000\n",
      "index: 102400, loss: 4681667.5000\n",
      "index: 102656, loss: 2090003.6250\n",
      "index: 102912, loss: 4217003.0000\n",
      "index: 103168, loss: 2082144.5000\n",
      "index: 103424, loss: 6093760.0000\n",
      "index: 103680, loss: 1399954.7500\n",
      "index: 103936, loss: 1611670.7500\n",
      "index: 104192, loss: 7499177.0000\n",
      "index: 104448, loss: 2876418.0000\n",
      "index: 104704, loss: 3102554.0000\n",
      "index: 104960, loss: 16381188.0000\n",
      "index: 105216, loss: 627787456.0000\n",
      "index: 105472, loss: 664016192.0000\n",
      "index: 105728, loss: 6867390.5000\n",
      "index: 105984, loss: 2294474.5000\n",
      "index: 106240, loss: 1586194.5000\n",
      "index: 106496, loss: 6613365.0000\n",
      "index: 106752, loss: 6842156.0000\n",
      "index: 107008, loss: 1454950.7500\n",
      "index: 107264, loss: 3834811.0000\n",
      "index: 107520, loss: 3349216.0000\n",
      "index: 107776, loss: 4799199.5000\n",
      "index: 108032, loss: 3080175.7500\n",
      "index: 108288, loss: 6438521.0000\n",
      "index: 108544, loss: 3648100.5000\n",
      "index: 108800, loss: 3338513.2500\n",
      "index: 109056, loss: 1309207.1250\n",
      "index: 109312, loss: 4475032.0000\n",
      "index: 109568, loss: 12486000.0000\n",
      "index: 109824, loss: 1991291.5000\n",
      "index: 110080, loss: 1715987.7500\n",
      "index: 110336, loss: 1596308.7500\n",
      "index: 110592, loss: 4892702.5000\n",
      "index: 110848, loss: 1894697.1250\n",
      "index: 111104, loss: 4250362.5000\n",
      "index: 111360, loss: 4922048.0000\n",
      "index: 111616, loss: 3465646.5000\n",
      "index: 111872, loss: 2184469.7500\n",
      "index: 112128, loss: 1356176.0000\n",
      "index: 112384, loss: 4806208.5000\n",
      "index: 112640, loss: 17219814.0000\n",
      "index: 112896, loss: 1141685248.0000\n",
      "index: 113152, loss: 191245984.0000\n",
      "index: 113408, loss: 11782620.0000\n",
      "index: 113664, loss: 32824690.0000\n",
      "index: 113920, loss: 46441344.0000\n",
      "index: 114176, loss: 49888632.0000\n",
      "index: 114432, loss: 9459994.0000\n",
      "index: 114688, loss: 1152893.5000\n",
      "index: 114944, loss: 746726.1875\n",
      "index: 115200, loss: 896868.2500\n",
      "index: 115456, loss: 2990149.0000\n",
      "index: 115712, loss: 1984568.5000\n",
      "index: 115968, loss: 1392763.1250\n",
      "index: 116224, loss: 1934110.8750\n",
      "index: 116480, loss: 3784940.0000\n",
      "index: 116736, loss: 5989554.5000\n",
      "index: 116992, loss: 3712165.2500\n",
      "index: 117248, loss: 1369354.7500\n",
      "index: 117504, loss: 847116.2500\n",
      "index: 117760, loss: 1534070.5000\n",
      "index: 118016, loss: 4685398.0000\n",
      "index: 118272, loss: 1025851008.0000\n",
      "index: 118528, loss: 9468396.0000\n",
      "index: 118784, loss: 3438411.0000\n",
      "index: 119040, loss: 3831657.5000\n",
      "index: 119296, loss: 10024210.0000\n",
      "index: 119552, loss: 14381463.0000\n",
      "index: 119808, loss: 2859291.0000\n",
      "index: 120064, loss: 6312236.0000\n",
      "index: 120320, loss: 4911146.0000\n",
      "index: 120576, loss: 5901339.0000\n",
      "index: 120832, loss: 5557907.0000\n",
      "index: 121088, loss: 4216774.0000\n",
      "index: 121344, loss: 8295668.5000\n",
      "index: 121600, loss: 5760219.0000\n",
      "index: 121856, loss: 3720371.5000\n",
      "index: 122112, loss: 3664390.7500\n",
      "index: 122368, loss: 4362652.5000\n",
      "index: 122624, loss: 3084002.7500\n",
      "index: 122880, loss: 4495357.5000\n",
      "index: 123136, loss: 3756855.7500\n",
      "index: 123392, loss: 2561025.5000\n",
      "index: 123648, loss: 2353939.7500\n",
      "index: 123904, loss: 1604112.8750\n",
      "index: 124160, loss: 1218530.8750\n",
      "index: 124416, loss: 2865759.2500\n",
      "index: 124672, loss: 1042794.8750\n",
      "index: 124928, loss: 1750874.7500\n",
      "index: 125184, loss: 2931077.0000\n",
      "index: 125440, loss: 2056026.2500\n",
      "index: 125696, loss: 2322475.7500\n",
      "index: 125952, loss: 1336015.5000\n",
      "index: 126208, loss: 1146824.8750\n",
      "index: 126464, loss: 987628.2500\n",
      "index: 126720, loss: 1513683.6250\n",
      "index: 126976, loss: 6464557.0000\n",
      "index: 127232, loss: 477216928.0000\n",
      "index: 127488, loss: 140255936.0000\n",
      "index: 127744, loss: 9389558.0000\n",
      "index: 128000, loss: 2316276.7500\n",
      "index: 128256, loss: 2882200.0000\n",
      "index: 128512, loss: 5639425.0000\n",
      "index: 128768, loss: 7900015.5000\n",
      "index: 129024, loss: 2311473.5000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "index: 129280, loss: 5345947.5000\n",
      "index: 129536, loss: 3072081.0000\n",
      "index: 129792, loss: 4227489.0000\n",
      "index: 130048, loss: 4295295.0000\n",
      "index: 130304, loss: 4332012.5000\n",
      "index: 130560, loss: 4636956.0000\n",
      "index: 130816, loss: 5481211.0000\n",
      "index: 131072, loss: 2071885.0000\n",
      "index: 131328, loss: 4298899.5000\n",
      "index: 131584, loss: 4098910.5000\n",
      "index: 131840, loss: 3905417.5000\n",
      "index: 132096, loss: 5066809.0000\n",
      "index: 132352, loss: 2797831.2500\n",
      "index: 132608, loss: 3887858.5000\n",
      "index: 132864, loss: 4821723.5000\n",
      "index: 133120, loss: 3663502.7500\n",
      "index: 133376, loss: 2259491.7500\n",
      "index: 133632, loss: 50285808.0000\n",
      "index: 133888, loss: 3832247.5000\n",
      "index: 134144, loss: 2560074.0000\n",
      "index: 134400, loss: 2301788.5000\n",
      "index: 134656, loss: 6069030.0000\n",
      "index: 134912, loss: 15496918.0000\n",
      "index: 135168, loss: 1949879.2500\n",
      "index: 135424, loss: 2788536.5000\n",
      "index: 135680, loss: 4276271.5000\n",
      "index: 135936, loss: 2872138.0000\n",
      "index: 136192, loss: 13347507.0000\n",
      "index: 136448, loss: 2096684.7500\n",
      "index: 136704, loss: 1714545024.0000\n",
      "index: 136960, loss: 9133767.0000\n",
      "index: 137216, loss: 2677021.7500\n",
      "index: 137472, loss: 2081990.8750\n",
      "index: 137728, loss: 5022598.0000\n",
      "index: 137984, loss: 5783160.0000\n",
      "index: 138240, loss: 2045056.2500\n",
      "index: 138496, loss: 3317694.2500\n",
      "index: 138752, loss: 2564621.0000\n",
      "index: 139008, loss: 2999492.5000\n",
      "index: 139264, loss: 2735088.5000\n",
      "index: 139520, loss: 3239539.2500\n",
      "index: 139776, loss: 1959196.6250\n",
      "index: 140032, loss: 2603830.0000\n",
      "index: 140288, loss: 1365034.2500\n",
      "index: 140544, loss: 2483580.0000\n",
      "index: 140800, loss: 1305041.2500\n",
      "index: 141056, loss: 1928974.6250\n",
      "index: 141312, loss: 2833364.0000\n",
      "index: 141568, loss: 3587723.2500\n",
      "index: 141824, loss: 2370051.5000\n",
      "index: 142080, loss: 1311335.7500\n",
      "index: 142336, loss: 1586467.5000\n",
      "index: 142592, loss: 938270.8125\n",
      "index: 142848, loss: 1096415.7500\n",
      "index: 143104, loss: 1428369.2500\n",
      "index: 143360, loss: 3025080.5000\n",
      "index: 143616, loss: 5153688.0000\n",
      "index: 143872, loss: 1903077.5000\n",
      "index: 144128, loss: 911822.4375\n",
      "index: 144384, loss: 787416.3125\n",
      "index: 144640, loss: 2695929.0000\n",
      "index: 144896, loss: 1322945.5000\n",
      "index: 145152, loss: 1217824.7500\n",
      "index: 145408, loss: 2339600.5000\n",
      "index: 145664, loss: 605498.8750\n",
      "index: 145920, loss: 665412096.0000\n",
      "index: 146176, loss: 8797686.0000\n",
      "index: 146432, loss: 2287966.7500\n",
      "index: 146688, loss: 2511197.7500\n",
      "index: 146944, loss: 8627394.0000\n",
      "index: 147200, loss: 5690621.0000\n",
      "index: 147456, loss: 3001453.7500\n",
      "index: 147712, loss: 7842879.0000\n",
      "index: 147968, loss: 6516602.0000\n",
      "index: 148224, loss: 6156804.0000\n",
      "index: 148480, loss: 1211149.7500\n",
      "index: 148736, loss: 3520998.2500\n",
      "index: 148992, loss: 2847698.7500\n",
      "index: 149248, loss: 3076452.2500\n",
      "index: 149504, loss: 1988056.5000\n",
      "index: 149760, loss: 2959110.7500\n",
      "index: 150016, loss: 10547226.0000\n",
      "index: 150272, loss: 3303582.0000\n",
      "index: 150528, loss: 2244212.7500\n",
      "index: 150784, loss: 3386449.7500\n",
      "index: 151040, loss: 1748680.5000\n",
      "index: 151296, loss: 3825640.2500\n",
      "index: 151552, loss: 1658195.8750\n",
      "index: 151808, loss: 3690740.0000\n",
      "index: 152064, loss: 4965601.5000\n",
      "index: 152320, loss: 1809610.5000\n",
      "index: 152576, loss: 1591422.7500\n",
      "index: 152832, loss: 3329737.5000\n",
      "index: 153088, loss: 1687046.0000\n",
      "index: 153344, loss: 10624174.0000\n",
      "index: 153600, loss: 2083367.6250\n",
      "index: 153856, loss: 890235776.0000\n",
      "index: 154112, loss: 8655899.0000\n",
      "index: 154368, loss: 5712501.0000\n",
      "index: 154624, loss: 54199712.0000\n",
      "index: 154880, loss: 29858958.0000\n",
      "index: 155136, loss: 28181178.0000\n",
      "index: 155392, loss: 11683308.0000\n",
      "index: 155648, loss: 24275128.0000\n",
      "index: 155904, loss: 25947004.0000\n",
      "index: 156160, loss: 24850204.0000\n",
      "index: 156416, loss: 16444813.0000\n",
      "index: 156672, loss: 5442865.5000\n",
      "index: 156928, loss: 7357563.0000\n",
      "index: 157184, loss: 14023528.0000\n",
      "index: 157440, loss: 4806980.5000\n",
      "index: 157696, loss: 7075512.0000\n",
      "index: 157952, loss: 4687187.5000\n",
      "index: 158208, loss: 6685662.5000\n",
      "index: 158464, loss: 5633764.5000\n",
      "index: 158720, loss: 5423434.0000\n",
      "index: 158976, loss: 1566031.1250\n",
      "index: 159232, loss: 3810573.5000\n",
      "index: 159488, loss: 4664691.0000\n",
      "index: 159744, loss: 2357116.7500\n",
      "index: 160000, loss: 5632783.0000\n",
      "index: 160256, loss: 1142501.2500\n",
      "index: 160512, loss: 3322709.5000\n",
      "index: 160768, loss: 10823158.0000\n",
      "index: 161024, loss: 3741296.5000\n",
      "index: 161280, loss: 3884013.5000\n",
      "index: 161536, loss: 2485065.5000\n",
      "index: 161792, loss: 8943264.0000\n",
      "index: 162048, loss: 3224604.5000\n",
      "index: 162304, loss: 2173779.5000\n",
      "index: 162560, loss: 7871024.0000\n",
      "index: 162816, loss: 906406400.0000\n",
      "index: 163072, loss: 176690512.0000\n",
      "index: 163328, loss: 13534722.0000\n",
      "index: 163584, loss: 2234615.2500\n",
      "index: 163840, loss: 3109247.5000\n",
      "index: 164096, loss: 4244477.5000\n",
      "index: 164352, loss: 8790343.0000\n",
      "index: 164608, loss: 3625201.2500\n",
      "index: 164864, loss: 3094190.0000\n",
      "index: 165120, loss: 6402541.5000\n",
      "index: 165376, loss: 5848827.0000\n",
      "index: 165632, loss: 4382687.5000\n",
      "index: 165888, loss: 1785608.7500\n",
      "index: 166144, loss: 3855599.2500\n",
      "index: 166400, loss: 6407690.0000\n",
      "index: 166656, loss: 3040852.5000\n",
      "index: 166912, loss: 3921001.5000\n",
      "index: 167168, loss: 6320675.0000\n",
      "index: 167424, loss: 3331886.5000\n",
      "index: 167680, loss: 5365474.0000\n",
      "index: 167936, loss: 4230731.5000\n",
      "index: 168192, loss: 2098778.2500\n",
      "index: 168448, loss: 3207488.5000\n",
      "index: 168704, loss: 4825542.5000\n",
      "index: 168960, loss: 3066997.2500\n",
      "index: 169216, loss: 4215169.5000\n",
      "index: 169472, loss: 1112343.5000\n",
      "index: 169728, loss: 6121678.0000\n",
      "index: 169984, loss: 4770368.0000\n",
      "index: 170240, loss: 4978451.0000\n",
      "index: 170496, loss: 16355468.0000\n",
      "index: 170752, loss: 3012260.0000\n",
      "index: 171008, loss: 2895288.5000\n",
      "index: 171264, loss: 4150469.0000\n",
      "index: 171520, loss: 3168848.0000\n",
      "index: 171776, loss: 12138820.0000\n",
      "index: 172032, loss: 1027285056.0000\n",
      "index: 172288, loss: 191402176.0000\n",
      "index: 172544, loss: 22597972.0000\n",
      "index: 172800, loss: 4487939.5000\n",
      "index: 173056, loss: 10051937.0000\n",
      "index: 173312, loss: 9786923.0000\n",
      "index: 173568, loss: 14459358.0000\n",
      "index: 173824, loss: 7139935.0000\n",
      "index: 174080, loss: 6055128.0000\n",
      "index: 174336, loss: 14102876.0000\n",
      "index: 174592, loss: 6119680.0000\n",
      "index: 174848, loss: 6314307.0000\n",
      "index: 175104, loss: 1191287.0000\n",
      "index: 175360, loss: 2055115.3750\n",
      "index: 175616, loss: 2970887.0000\n",
      "index: 175872, loss: 2330836.7500\n",
      "index: 176128, loss: 2911369.0000\n",
      "index: 176384, loss: 3890034.5000\n",
      "index: 176640, loss: 4042936.0000\n",
      "index: 176896, loss: 2536242.0000\n",
      "index: 177152, loss: 2792740.0000\n",
      "index: 177408, loss: 2154302.0000\n",
      "index: 177664, loss: 1836958.7500\n",
      "index: 177920, loss: 3098013.5000\n",
      "index: 178176, loss: 1487192.7500\n",
      "index: 178432, loss: 4081797.0000\n",
      "index: 178688, loss: 860765.2500\n",
      "index: 178944, loss: 1793376.0000\n",
      "index: 179200, loss: 3451076.5000\n",
      "index: 179456, loss: 2722809.5000\n",
      "index: 179712, loss: 1744943.5000\n",
      "index: 179968, loss: 1927129.8750\n",
      "index: 180224, loss: 4332571.5000\n",
      "index: 180480, loss: 2273490.5000\n",
      "index: 180736, loss: 1455441.6250\n",
      "index: 180992, loss: 5668631.0000\n",
      "index: 181248, loss: 576623744.0000\n",
      "index: 181504, loss: 175400832.0000\n",
      "index: 181760, loss: 15984842.0000\n",
      "index: 182016, loss: 2956654.0000\n",
      "index: 182272, loss: 6029737.5000\n",
      "index: 182528, loss: 15190298.0000\n",
      "index: 182784, loss: 42129568.0000\n",
      "index: 183040, loss: 6499237.0000\n",
      "index: 183296, loss: 2554014.5000\n",
      "index: 183552, loss: 5473760.0000\n",
      "index: 183808, loss: 2884796.0000\n",
      "index: 184064, loss: 2987290.7500\n",
      "index: 184320, loss: 1009940.4375\n",
      "index: 184576, loss: 1575525.0000\n",
      "index: 184832, loss: 3671462.5000\n",
      "index: 185088, loss: 2135064.0000\n",
      "index: 185344, loss: 2840377.5000\n",
      "index: 185600, loss: 4566219.0000\n",
      "index: 185856, loss: 2913760.0000\n",
      "index: 186112, loss: 2510688.5000\n",
      "index: 186368, loss: 2711717.0000\n",
      "index: 186624, loss: 2402529.5000\n",
      "index: 186880, loss: 1420258.0000\n",
      "index: 187136, loss: 2025988.7500\n",
      "index: 187392, loss: 1412296.2500\n",
      "index: 187648, loss: 7492717.0000\n",
      "index: 187904, loss: 953584.6250\n",
      "index: 188160, loss: 1734677.8750\n",
      "index: 188416, loss: 3704388.2500\n",
      "index: 188672, loss: 2286866.5000\n",
      "index: 188928, loss: 17484082.0000\n",
      "index: 189184, loss: 1467224.0000\n",
      "index: 189440, loss: 4447935.0000\n",
      "index: 189696, loss: 1880265.7500\n",
      "index: 189952, loss: 1297514.2500\n",
      "index: 190208, loss: 3587297.2500\n",
      "index: 190464, loss: 905959.7500\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Done Training. End of data stream."
     ]
    }
   ],
   "source": [
    "check_error = EarlyStop(20)\n",
    "\n",
    "cur_loss = sys.float_info.max # starting with largest loss possible\n",
    "raw_batch = data.next_batch()\n",
    "training = check_error(raw_batch, cur_loss)\n",
    "\n",
    "outfile = open('results', 'w')\n",
    "\n",
    "while training:\n",
    "    data_dict = split_batch(raw_batch, dataspecs)\n",
    "    targets = {'target_' + name : data_dict[name] for name in loss_feats}\n",
    "    data_dict.update(targets)\n",
    "    cur_loss, feat_loss = model.eval(data_dict, eval_tensors)\n",
    "    model.train_step(data_dict)\n",
    "    \n",
    "    write_results(data_dict, feat_loss, outfile)\n",
    "    print('index: %s, loss: %.4f' % (data.index, cur_loss))\n",
    "    raw_batch = data.next_batch()\n",
    "    training = check_error(raw_batch, cur_loss)\n",
    "    \n",
    "outfile.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
